Teleassistance data prioritization for self-driving vehicles

ABSTRACT

A system can analyze a live sensor view of a self-driving vehicle (SDV) in accordance with a safety threshold to detect objects of interest along a route and classify each detected object of interest. When the safety threshold is not met, the system can transmit a teleassistance inquiry using LIDAR data to a backend computing system. When a certainty threshold is not met for an object of interest, the system can transmit a different teleassistance inquiry using image data to the backend computing system.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/452,431, titled Teleassistance Data Prioritization for Self-DrivingVehicles, and filed on Mar. 7, 2017; which is hereby incorporated byreference in its entirety.

BACKGROUND

On-board features of modern vehicles can have the effect of distractingthe driver from safe operation of the vehicle. Such on-board featurescan include BLUETOOTH telephony, radio, CD, or MP3 audio players,on-board DVD or MP4 video players, mapping and routing features, andon-board voice text messaging or SMS features that enable drivers toread and response to text messages while, at the same time, attemptingto safely operate the vehicle in traffic. Such on-board features mayfurther include on-demand television, social media and other contentbrowsing features, calendar scheduling, and voice and videoconferencing. As network technology progresses, modern vehicles willcontinue to evolve into “smart” vehicles capable of providing the userwith evolutions of the above-mentioned on-board features that contributeto visual, manual, and cognitive distraction.

At the same time, vehicle manufacturers continue to provide evolvingsafety features that include not only structural and mechanicalcharacteristics (e.g., crumple structures, carbon fiber materials,etc.), but also software-based driving aids that support the driver, forexample, in lane-keeping, following, preemptive braking, and collisiondetection and avoidance.

Autonomous vehicle technology seeks to circumvent manufacturer conflictbetween providing desirable on-board features and maintaining driverawareness for safe driving. Nevertheless, the human brain hassignificant advantages over current autonomous driving systems, beingcapable of rapid and intelligent perception, thought, anddecision-making. In order to progress from current autonomouscapabilities to full autonomy and acceptable trustworthiness in all roadconditions and environments, leveraging human cognition may beadvantageous in assisting and intervening autonomous vehicle operationsin various scenarios.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure herein is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements, and in which:

FIG. 1 is a block diagram illustrating an example self-driving vehicleoperated by a control system implementing a teleassistance module, asdescribed herein;

FIG. 2 is a block diagram illustrating an example teleassistance moduleutilized in connection with a self-driving vehicle, according toexamples described herein;

FIG. 3 shows an example of an autonomously controlled self-drivingvehicle utilizing sensor data to navigate an environment in accordancewith example implementations;

FIG. 4 shows an example self-driving vehicle executing a resolutionresponse from a backend teleassistance operator, in accordance withexample implementations;

FIGS. 5A and 5B are flow charts describing example methods of operatinga self-driving vehicle using teleassistance, according to examplesdescribed herein;

FIG. 6 is another flow chart describing an example method of operating aself-driving vehicle using teleassistance, according to examplesdescribed herein;

FIG. 7 is a block diagram illustrating a computer system for aself-driving vehicle upon which examples described herein may beimplemented; and

FIG. 8 is a block diagram illustrating a computer system for a backenddatacenter upon which example transport systems described herein may beimplemented.

DETAILED DESCRIPTION

A self-driving vehicle (SDV) can include a sensor suite to generate alive sensor view of a surrounding area of the SDV and acceleration,braking, and steering systems autonomously operated by a control system.In various implementations, the control system can dynamically analyzethe sensor view of the surrounding area and a road network map (e.g., ahighly detailed localization map) in order to autonomously operate theacceleration, braking, and steering systems along a current route to adestination.

In certain examples, the SDV can include a communication interface toenable the control system to communicate to a backend transport system.In one aspect, the backend transport system can be implemented throughone or more datacenters and can manage on-demand transportation servicesfor a fleet of SDVs operating throughout a given region (e.g., ametroplex such as the San Francisco Bay metropolitan area). In someexamples, as the SDV operates throughout the given region, the SDV cantransmit and receive communications for the backend transport system.These communications can comprise transport instructions to rendezvouswith a requesting user at a pick-up location, location data indicatingsuch pick-up locations and destinations, routing data indicating anoptimal route to the pick-up location and then onwards to thedestination, and routing updates indicating dynamic changes to trafficconditions along a particular route.

In various implementations, the SDV can also communicate teleassistancedata to the backend transport system, or a remote teleassistance system.For example, the control system of the SDV can include perception andprediction engines that can make decisions with regard to acceleration,braking, and steering inputs in accordance with confidence metrics inboth detecting respective fields of view and classifying detectedobjects in those fields of view. Based on a detected anomaly in the livesensor view, the control system can transmit a teleassistance inquiry tothe transport system, which can be processed by human teleassistanceoperators. In certain aspects, the control system can transmit sensordata in conjunction with the teleassistance inquiry in accordance with adata prioritization scheme. Additionally or alternatively, the controlsystem can parse, encode, and/or compress the sensor data prior totransmitting or streaming the data to teleassistance operators. Infurther examples, the control system can specify an anomalous objectwith a bounding box, which may be encoded at a higher quality than aremainder of the sensor data.

In order to ensure that the teleassistance data is properly transmitted,the control system can transmit data in accordance with a prioritizationscheme. For example, the SDV may comprise multiple communication modulescommunicating with the transport system using any number ofcommunications standards and/or protocols. Some communication modulesmay be more reliable than others in general, or depending on thelocation of the SDV in the given region. At a high level, theprioritization scheme can cause the control system to prioritizeteleassistance communications over other communications (e.g., as ahighest priority data transmission), and at a lower level, canprioritize certain types of sensor data depending on the type and natureof the anomaly. Specifically, the SDV can communicate location pings andother periodic data to the backend transport system, where sometransmissions may require retransmission or may be lost outright. Thecontrol system can monitor the bandwidth and performance of the SDV'scommunication modules to ensure that any teleassistance communicationsbe transmitted over the most reliable and/or highest bandwidthcommunication module(s). Furthermore, the control system can categorizethe teleassistance inquiry as, for example, a detection issue or anobject classification issue. In one aspect, for detection issues (e.g.,an obstruction in the sensor view) the control system can prioritizeLIDAR data for transmission with the teleassistance inquiry, and forclassification issues (e.g., an indeterminate object) the control systemcan prioritize image or video data.

In still further implementations, the control system may have difficultyeither resolving objects in the live sensor view, or operating thecontrol mechanisms of the SDV (e.g., the acceleration, braking, and/orsteering systems). Such difficulties may arise from fault or failedconditions in the sensors, the computer stack of the vehicle, or from amechanical system of the vehicle itself (e.g., a tire with low airpressure affecting steering). In these implementations, the controlsystem can transmit diagnostics data to enable the teleassistanceoperator to provide a resolution.

In many aspects, the teleassistance inquiry can seek assistance frombackend operators in resolving detection difficulties, such as when acritical occlusion exists in the live sensor view, or when an object ofinterest is not readily classifiable by the control system. It iscontemplated that on-board computational capabilities of self-drivingvehicles may be inferior to the computing power of backend datacenters.Accordingly, the backend teleassistance operators may be human or may beteleassistance computers or virtual machines executing instruction setsfor detecting and classifying objects in road environments, or can rundeep learning models specific to those tasks. According to examplesprovided herein, the control system can receive a resolution responsefrom the backend teleassistance operator based on the teleassistanceinquiry. The resolution response can comprise data classifying theunresolved object of interest, or can include an instruction of how toproceed when a detection anomaly exists. The control system may thenproceed along the current route (or an alternate route) in accordancewith the resolution response.

In certain aspects, the teleassistance operator may require additionaldata to resolve a detected anomaly. For example, localizationinformation corresponding to the location and orientation of the SDV inthe given region may provide context to the teleassistance operatorgiven a current overall traffic situation local to the SDV. In somescenarios, diagnostics information of the SDV and/or the SDV's sensorsystems can provide context to the teleassistance operator when, forexample, a malfunction or fault has occurred with an SDV system (e.g., afailed tire, a fault condition on a sensor, debris occluding the sensorview, etc.). Providing this additional data can aid the teleassistanceoperator in assisting the SDV and constructing or otherwise generatingthe resolution package to enable the SDV to proceed.

Among other benefits, the examples described herein achieve a technicaleffect of providing dynamic resolution of anomalies for a self-drivingvehicle (SDV) when operating on public roads and highways. Currenttechnologies typically involve the SDV pulling over or stopping whenanomalous conditions exist, or an on-board human engineer being providedwith alerts and resolving them on-the-fly, or even an alert human backupdriver sitting in the driver's seat of the SDV, ready to take overmanual control of the SDV during emergency events. Prioritization ofteleassistance data and specialized encoding and/or duplication ofteleassistance inquiries and streamed sensor data can provide forimproved resolution of detected anomalies with reduced impact onbandwidth usage.

As used herein, a computing device refers to devices corresponding todesktop computers, cellular devices or smartphones, personal digitalassistants (PDAs), laptop computers, tablet devices, virtual reality(VR) and/or augmented reality (AR) devices, wearable computing devices,television (IP Television), etc., that can provide network connectivityand processing resources for communicating with the system over anetwork. A computing device can also correspond to custom hardware,in-vehicle devices, or on-board computers, etc. The computing device canalso operate a designated application configured to communicate with thenetwork service.

One or more examples described herein provide that methods, techniques,and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmatically,as used herein, means through the use of code or computer-executableinstructions. These instructions can be stored in one or more memoryresources of the computing device. A programmatically performed step mayor may not be automatic.

One or more examples described herein can be implemented usingprogrammatic modules, engines, or components. A programmatic module,engine, or component can include a program, a sub-routine, a portion ofa program, or a software component or a hardware component capable ofperforming one or more stated tasks or functions. As used herein, amodule or component can exist on a hardware component independently ofother modules or components. Alternatively, a module or component can bea shared element or process of other modules, programs or machines.

Some examples described herein can generally require the use ofcomputing devices, including processing and memory resources. Forexample, one or more examples described herein may be implemented, inwhole or in part, on computing devices such as servers, desktopcomputers, cellular or smartphones, personal digital assistants (e.g.,PDAs), laptop computers, virtual reality (VR) or augmented reality (AR)computers, network equipment (e.g., routers) and tablet devices. Memory,processing, and network resources may all be used in connection with theestablishment, use, or performance of any example described herein(including with the performance of any method or with the implementationof any system).

Furthermore, one or more examples described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing examples disclosed herein can be carriedand/or executed. In particular, the numerous machines shown withexamples of the invention include processors and various forms of memoryfor holding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as those carried on smartphones, multifunctional devices ortablets), and magnetic memory. Computers, terminals, network enableddevices (e.g., mobile devices, such as cell phones) are all examples ofmachines and devices that utilize processors, memory, and instructionsstored on computer-readable mediums. Additionally, examples may beimplemented in the form of computer-programs, or a computer usablecarrier medium capable of carrying such a program.

As provided herein, the terms “autonomous vehicle” (AV) or “self-drivingvehicle” (SDV) may be used interchangeably to describe any vehicleoperating in a state of autonomous control with respect to acceleration,steering, and braking. Different levels of autonomy may exist withrespect to AVs and SDVs. For example, some vehicles may enableautonomous control in limited scenarios, such as on highways. Moreadvanced AVs and SDVs can operate in a variety of traffic environmentswithout any human assistance. Accordingly, an “AV control system” canprocess sensor data from the AV or SDV's sensor array, and modulateacceleration, steering, and braking inputs to safely drive the AV or SDValong a given route.

System Description

FIG. 1 is a block diagram illustrating an example self-driving vehicleoperated by a control system implementing a teleassistance module, asdescribed herein. In an example of FIG. 1, a control system 120 canautonomously operate the SDV 100 in a given geographic region for avariety of purposes, including transport services (e.g., transport ofhumans, delivery services, etc.). In examples described, the SDV 100 canoperate without human control. For example, the SDV 100 can autonomouslysteer, accelerate, shift, brake, and operate lighting components. Somevariations also recognize that the SDV 100 can switch between anautonomous mode, in which the SDV control system 120 autonomouslyoperates the SDV 100, and a manual mode in which a driver takes overmanual control of the acceleration system 172, steering system 174,braking system 176, and lighting and auxiliary systems 178 (e.g.,directional signals and headlights).

According to some examples, the control system 120 can utilize specificsensor resources in order to autonomously operate the SDV 100 in avariety of driving environments and conditions. For example, the controlsystem 120 can operate the SDV 100 by autonomously operating thesteering, acceleration, and braking systems 172, 174, 176 of the SDV 100to a specified destination 137. The control system 120 can performvehicle control actions (e.g., braking, steering, accelerating) androute planning using sensor information, as well as other inputs (e.g.,transmissions from remote or local human operators, networkcommunication from other vehicles, etc.).

In an example of FIG. 1, the control system 120 includes computationalresources (e.g., processing cores and/or field programmable gate arrays(FPGAs)) which operate to process sensor data 115 received from a sensorsystem 102 of the SDV 100 that provides a sensor view of a road segmentupon which the SDV 100 operates. The sensor data 115 can be used todetermine actions which are to be performed by the SDV 100 in order forthe SDV 100 to continue on a route to the destination 137. In somevariations, the control system 120 can include other functionality, suchas wireless communication capabilities using a communication interface135, to send and/or receive wireless communications over one or morenetworks 185 with one or more remote sources. In controlling the SDV100, the control system 120 can generate commands 158 to control thevarious control mechanisms 170 of the SDV 100, including the vehicle'sacceleration system 172, steering system 157, braking system 176, andauxiliary systems 178 (e.g., lights and directional signals).

The SDV 100 can be equipped with multiple types of sensors 102 which cancombine to provide a computerized perception of the space and thephysical environment surrounding the SDV 100. Likewise, the controlsystem 120 can operate within the SDV 100 to receive sensor data 115from the collection of sensors 102 and to control the various controlmechanisms 170 in order to autonomously operate the SDV 100. Forexample, the control system 120 can analyze the sensor data 115 togenerate low level commands 158 executable by the acceleration system172, steering system 157, and braking system 176 of the SDV 100.Execution of the commands 158 by the control mechanisms 170 can resultin throttle inputs, braking inputs, and steering inputs thatcollectively cause the SDV 100 to operate along sequential road segmentsto a particular destination 137.

In more detail, the sensors 102 operate to collectively obtain a sensorview for the SDV 100 (e.g., in a forward operational direction, orproviding a 360 degree sensor view), and further to obtain situationalinformation proximate to the SDV 100, including any potential hazards orobstacles. By way of example, the sensors 102 can include multiple setsof camera systems 101 (video cameras, stereoscopic cameras or depthperception cameras, long range monocular cameras), LIDAR systems 103,one or more radar systems 105, and various other sensor resources suchas sonar, proximity sensors, infrared sensors, and the like. Accordingto examples provided herein, the sensors 102 can be arranged or groupedin a sensor system or array (e.g., in a sensor pod mounted to the roofof the SDV 100) comprising any number of LIDAR, radar, monocular camera,stereoscopic camera, sonar, infrared, or other active or passive sensorsystems.

Each of the sensors 102 can communicate with the control system 120utilizing a corresponding sensor interface 110, 112, 114. Each of thesensor interfaces 110, 112, 114 can include, for example, hardwareand/or other logical components which are coupled or otherwise providedwith the respective sensor. For example, the sensors 102 can include avideo camera and/or stereoscopic camera system 101 which continuallygenerates image data of the physical environment of the SDV 100. Thecamera system 101 can provide the image data for the control system 120via a camera system interface 110. Likewise, the LIDAR system 103 canprovide LIDAR data to the control system 120 via a LIDAR systeminterface 112. Furthermore, as provided herein, radar data from theradar system 105 of the SDV 100 can be provided to the control system120 via a radar system interface 114. In some examples, the sensorinterfaces 110, 112, 114 can include dedicated processing resources,such as provided with field programmable gate arrays (FPGAs) which can,for example, receive and/or preprocess raw image data from the camerasensor.

In general, the sensor systems 102 collectively provide sensor data 115to a perception/prediction engine 140 of the control system 120. Theperception/prediction engine 140 can access a database 130 comprisingstored localization maps 132 of the given region in which the SDV 100operates. The localization maps 132 can comprise highly detailed groundtruth data of each road segment of the given region. For example, thelocalization maps 132 can comprise prerecorded data (e.g., sensor dataincluding image data, LIDAR data, and the like) by specialized mappingvehicles or other SDVs with recording sensors and equipment, and can beprocessed to pinpoint various objects of interest (e.g., trafficsignals, road signs, and other static objects). As the SDV 100 travelsalong a given route, the perception/prediction engine 140 can access acurrent localization map 133 of a current road segment to compare thedetails of the current localization map 133 with the sensor data 115 inorder to detect and classify any objects of interest, such as movingvehicles, pedestrians, bicyclists, and the like.

In various examples, the perception/prediction engine 140 candynamically compare the live sensor data 115 from the SDV's sensorsystems 102 to the current localization map 133 as the SDV 100 travelsthrough a corresponding road segment. The perception/prediction engine140 can flag or otherwise identify any objects of interest in the livesensor data 115 that can indicate a potential hazard. In accordance withmany examples, the perception/prediction engine 140 can output aprocessed sensor view 141 indicating such objects of interest to avehicle control module 155 of the SDV 100. In further examples, theperception/prediction engine 140 can predict a path of each object ofinterest and determine whether the SDV control system 120 should respondor react accordingly. For example, the perception/prediction engine 140can dynamically calculate a collision probability for each object ofinterest, and generate event alerts 151 if the collision probabilityexceeds a certain threshold. As described herein, such event alerts 151can be processed by the vehicle control module 155 that generatescontrol commands 158 executable by the various control mechanisms 170 ofthe SDV 100, such as the SDV's acceleration, steering, and brakingsystems 172, 174, 176.

On a higher level, the SDV control system 120 can include a routeplanning engine 160 that provides the vehicle control module 155 with aroute plan 139 and a travel trajectory 126 along a current route 139 toa destination 137. The current route 139 may be determined by a backendtransport system 190, or may be determined by the SDV 100 via access toa local or external mapping service. In some aspects, the SDV 100 caninclude a user interface 145, such as a touch-screen panel or speechrecognition features, which can enable a passenger to input adestination 137. Additionally or alternatively, the SDV control system120 can include a communication interface 135 providing the SDV 100 withconnectivity to one or more networks 185. In such implementations, theSDV 100 may communicate with a backend transport system 190 that managesrouting of any number of SDVs operating throughout a given region toprovide transportation services to requesting riders. Thus, the routeplanning engine 160 may receive the destination 137 from the backendtransport system 190 over the network(s) 185 in order to plan a currentroute 139 for the SDV 100.

In mapping the current route 139, the route planning engine 160 cangenerally utilize an on-board mapping engine or an external mappingservice by transmitting map calls over the network(s) 185 in order todetermine a most optimal route plan 139 from a current location of theSDV 100 to the destination 137. This route plan 139 may be determinedbased on distance, time, traffic conditions, additional pick-ups (e.g.,for carpooling services), and the like. For each successive road segmenton which the SDV 100 travels, the route planning engine 160 can providetrajectory data 126 to the vehicle control module 155 to enable thevehicle control module 155 to operate the SDV 100 safely to the nextroad segment or the destination 137. For example, the trajectory data126 can indicate that the vehicle control module 155 must change lanesor make a turn in order to proceed to the next road segment along thecurrent route plan 139.

According to examples provided herein, the vehicle control module 155can utilize the trajectory data 126, the processed sensor view 141, andevent alerts 151 to autonomously operate the control mechanisms 170 ofthe SDV 100. As a basic example, to make a simple turn based on thetrajectory data 126, the vehicle control module 155 can generate controlcommands 158 that cause the lights and auxiliary systems 178 of the SDV100 to activate the appropriate directional signal, the braking system176 to slow the SDV 100 down for the turn, the steering system 174 tosteer the SDV 100 into the turn, and the acceleration system 172 topropel the SDV 100 when exiting the turn. In further examples, eventalerts 151 may indicate potential hazards such as a pedestrian crossingthe road, a nearby bicyclist, obstacles on the road, a constructionarea, proximate vehicles, an upcoming traffic signal and signal state,and the like. The vehicle control module 155 can respond to each eventalert 151 on a lower level while, on a higher level, operating the SDV100 along the determined route plan 139 using the processed sensor view141.

In various examples, the backend transport system 190 can manage highlevel routing for a fleet of self-driving vehicles operating throughouta given region. For example, the transport system 190 can receivepick-up requests from requesting users or riders seeking on-demandtransportation services, such as the various on-demand transportationarrangement services provided by UBER TECHNOLOGIES Inc., of SanFrancisco, Calif. In such examples, when a requesting user submits apick-up request to the transport system 190, the transport system 190can identify a set of proximate vehicles (e.g., SDVs and/or human-drivenvehicles) with respect to a pick-up location for the requesting user. Invarious aspects, the transport system 190 can receive location pings 121from location-based resources (e.g., GPS units 122) of SDVs and/ordriver devices (e.g., smartphones executing a designated driverapplication) to identify their locations. The transport system 190 maythen select a most optimal SDV 100 amongst the proximate set andtransmit an instruction to service the pick-up request. In certainexamples, once a confirmation is received from the SDV 100, thetransport system 190 can transmit a route update indicating the pick-uplocation, and in some aspects, an optimal route from the currentlocation of the driver or SDV 100 to the pick-up location.

According to examples described herein, the control system 120 canfurther include a teleassistance module 125 to enable teleassistanceoperators 199 to aid the SDV 100 in progressing along the route plan 139when an anomaly is detected. As provided herein, the anomaly cancomprise a detection anomaly in which the control system 120 hasdifficulty detecting objects (e.g., due to an occlusion), anidentification or classification anomaly in which theperception/prediction engine 140 has difficulty classifying detectedobjects, or a fault anomaly corresponding to a diagnostics fault orfailure of a component of the SDV 100, such as a computer, a mechanicalcomponent, or a sensor.

In various implementations, when an anomaly is detected or determined,the perception/prediction engine 140 can submit a teleassistance inquiry143 to the teleassistance module 125. The teleassistance module 125 cantreat the inquiry 143 based on the type of anomaly to, for example,compile sensor data 115, prioritize certain types of sensor data 115,encode the sensor data 115 at different rates or qualities, specify ananomalous object in the sensor data 115 (e.g., using a bounding box),and/or incorporating telemetry, diagnostic data, and/or localizationdata (e.g., position and orientation of the SDV 100) with the inquiry143. The teleassistance module 125 may then transmit the inquiry 143 tothe teleassistance operator(s) 199 for processing. As described herein,the teleassistance operators 199 may be human operators trained toanalyze and resolve anomalies. Further, the teleassistance operators 199can be associated with the backend transport system 190, or may compriseone or more separate entities.

The teleassistance operators 199 may process the teleassistance inquiry143, determine a resolution to the inquiry 143, and transmit aresolution package 192 back to the communication interface 135 of theSDV 100. The resolution package 192 can comprise a simple identificationof an anomalous object, or can comprise instructions for proceeding,such as instructions for the SDV 100 to pull over immediately,instructions to proceed to a nearest service station, or instructions toignore a particular fault. Detailed description of the teleassistancemodule 125 in communication with the teleassistance operators 199 isprovided below with respect to FIG. 2.

FIG. 2 is a block diagram illustrating an example teleassistance moduleutilized in connection with a self-driving vehicle, according toexamples described herein. In the below description of FIG. 2, certainlogical blocks of the SDV 200 of FIG. 2 may be omitted or combined fromthe SDV 1000 of FIG. 1 for the sake of illustration and emphasis on thefeatures and functions of the teleassistance module 225. Thus, one ormore features as shown and described with respect to FIG. 1 may beincluded as a component of FIG. 2. Furthermore, the teleassistancemodule 225 as shown and described with respect to FIG. 2 can comprisethe teleassistance module 125 of the SDV control system 120 as shown anddescribed with respect to FIG. 1.

Referring to FIG. 2, the SDV 200 can include a sensor suite 210comprising a number of individual sensor systems, such as a camerasystem and LIDAR system, and any number of additional sensors.Furthermore, each sensor system can include any number of individualsensors. For example, the camera system can include any number ofindividual cameras (e.g., monocular cameras and stereoscopic cameras)each having an individual field of view that combined to create the liveimage view of the SDV 200. Furthermore, the LIDAR system can include anynumber of individual LIDAR sensors each providing a LIDAR view. TheLIDAR system and the camera system can combine to create a completesensor view for the SDV control system 205 to operate the SDV's controlmechanisms 295.

In doing so, the sensor data 212 can be processed by a perception engine215, which can analyze the sensor data 212 for objects of interest, suchas any pedestrians, bicyclists, other vehicles, traffic signals, roadsigns, and the like. For example, the perception engine 215 candynamically compare the sensor data 212 with sequential, on-boardlocalization maps of the given region in which the SDV 200 operates, asdescribed herein. The perception engine 215 can identify or otherwisemark the objects of interest, and provide object of interest data 219 toa prediction engine 220, which can predict whether each of the objectsof interest present a hazard. For example, the prediction engine 220 canclassify each the objects of interest, determine a path and/or velocityfor each of the objects of interest, and determine whether the SDV 200should proceed with caution, brake, weave, accelerate, change lanes, andthe like.

In various examples, the prediction engine 220 and the perception engine215 can combine to provide control inputs 221 to a vehicle controller280. The control inputs 221 can comprise acceleration, braking, andsteering inputs, as well as inputs to engage the auxiliary system, thehorn, windshield wipers, etc. The control inputs 221 can be processed bythe vehicle controller 280 to generate control commands 282 for each ofthe relevant control mechanisms 295 of the SDV 200. As provided herein,the SDV 200 can further include a mapping/routing engine 285, which canprovide the vehicle controller 280 with a route plan 287 to adestination. Thus, the vehicle controller 280 can proceed along thecurrent route plan 287 on a higher level, and utilize the control inputs221 from the perception and prediction engines 215, 220 to operate thecontrol mechanisms 295 of the SDV 200. The control commands 282 can beexecutable be the signaling, acceleration, braking, and steering systemsof the SDV 200 that comprise the control mechanisms 295.

As provided herein, the perception engine 215 can identify a detectionanomaly, such as debris on a camera sensor or an occlusion in the sensorview. If the detection anomaly is critical or otherwise prevents theperception engine 215 from enabling the SDV 200 to continue along theroute plan 287, the perception engine 215 can provide a detectioninquiry 217 to a teleassistance module 225 of the SDV 200. For example,the perception engine 215 can analyze the sensor data 212, or livesensor view of the SDV 200, in accordance with a safety standard. Assuch, if a safety perception threshold is not met, the perception engine215 can seek assistance from the teleassistance module 225. Furthermore,in some aspects, each of the sensor systems of the sensor suite 210 maybe associated with a safety standard. For example, if a field of view ofa camera is occluded, or if excessive processing resources of theperception engine 215 are required to resolve the sensor data 212 fromthe individual sensor systems, the perception engine 215 may submit adetection inquiry 217 specifying the detection anomaly.

As further provided herein, if the prediction engine 220 cannot classifya detected object or otherwise indicates a detected object asindeterminate, the prediction engine 220 may seek assistance from theteleassistance module 225. Specifically, the prediction engine 220 cansubmit an object inquiry 222 to the teleassistance module 225 in orderto resolve the indeterminate object. In certain examples, the predictionengine 220 can attempt to classify objects of interest in accordancewith a certainty threshold. If the certainty threshold is not met, thenthe prediction engine 220 can transmit the object inquiry 22 to theteleassistance module 225.

The teleassistance module 225 can include a perception interface 230 toreceive the detection inquiries 217 from the perception engine 215 andthe object inquiries 222 from the prediction engine 220. In conjunctionwith the inquiries 217, 222, the perception interface 230 can receivesensor data 212 corresponding to the inquiries (e.g., live or recordedsensor view data comprising the data from one or more of the sensorsystems). The perception interface 230 may then provide a local view 232specific to the detection anomaly or object classification anomaly to adata parsing engine 235. The local view 232 corresponding to thedetected anomaly can comprise data from individual sensor systemspertinent to the detected anomaly. For example, if the anomaly comprisesan occlusion in a forward direction, the local view 232 can comprisesensor data 212 from forward sensors of the sensor suite 210 (e.g., amain LIDAR sensor and a number of forward facing cameras). As anotherexample, if the detected anomaly corresponds to an indeterminate object,then the local view 232 can comprise sensor data 212 from sensorsdetecting the indeterminate object (e.g., a specified LIDAR and/or oneor more cameras).

The data parsing engine 235 can parse the local view 232 based on thespecific anomaly. For example, if the detected anomaly comprises adetection anomaly, then the data parsing engine 235 may parse the localsensor view 232 to include only LIDAR data or data from specific sensorsthat may aid in resolving the detected anomaly. Thus, in response toidentifying an occlusion in the live sensor view, the data parsingengine 235 can select sensor data 212 from the relevant sensors and/orexclude sensor data 212 from irrelevant sensors. In further examples, ifthe prediction engine 220 submits an object inquiry 222, the parsingengine 235 can bound the indeterminate object in image data fromselected sensors, and can provide the parsed data 237—with theindeterminate object bounded by, for example, a bounding box—to a dataencoder 240.

In various examples, the data encoder 240 can determine the quality atwhich the parsed data 237 is to be compressed or encoded fortransmission to the teleassist system 265. The teleassistance module 225can include a communications interface 250 coupled to a communicationsarray 255 of the SDV 200. The communications interface 250 can receivebandwidth data 257 from each communication module of the communicationsarray 255. In the example shown in FIG. 2, the communications array 255includes communications modules 1, 2, 3, and 4. However, inimplementation, the communications array 255 can comprise any number andcombination of communications modules configured for wirelesscommunication using any wireless communications protocol. Suchcommunications protocols can include third generation (3G), fourthgeneration (4G), long-term evolution (LTE), Wireless Gigabit Alliance(WiGig), WiMax, Wi-Fi, dedicated short range communications (DSRC),BLUETOOTH, and other like communications standards.

In some aspects, the data encoder 240 can determine the manner in whichthe parsed data 237 is to be compressed or encoded based on thebandwidth data 257 from the communications modules of the communicationsarray 255. In variations, the data encoder 240 can encode the parseddata 237 based on the objects of interest in the data 237. For example,the data encoder 240 can encode the bounded indeterminate object at ahigher bitrate than the remainder of the parsed data 237. The dataencoder 240 may then submit the encoded data 242 to a duplication module245, which can determine whether to duplicate transmission of theencoded data 242 based on, for example, the bandwidth data 257 from thecommunications array 255.

In various aspects, the duplication module 245 can determine that theavailable bandwidth is sufficient on a selected one of thecommunications modules, and generate an inquiry data package 247 fortransmission over the highest bandwidth communications module, oranother communications module having sufficient bandwidth. In oneexample, the communications array 255 may have communications modulesfrom different wireless communications service providers (e.g., multiple4G LTE providers), and may select one or the other based on such factorsas available bandwidth, historical reliability, cost, location, etc. Incertain conditions, the duplication module 245 can opt to transmit theinquiry data package 247 over multiple communications modules of thecommunications array 255 (e.g., multiple modems of different serviceproviders) in order to ensure adequate data transmission or streaming.

In one example, the duplication module 245 can select communicationsmodules from a plurality of wireless communication modules of thecommunications array 255, which enable communication with theteleassistance system 265 via a plurality of wireless communicationproviders (e.g., using multiple separate modems). In response toidentifying the detected anomaly in the live sensor view, theduplication module 245 can determine a highest bandwidth communicationmodule from the plurality of wireless communication modules. In someaspects, the duplication module 245 may transmit the teleassistanceinquiry 247 to the teleassistance system 265 using the highest bandwidthcommunication module. In variations, and in accordance with the dataprioritization scheme, the duplication module 245 can duplicatetransmission of the teleassistance inquiry 247 by transmitting theteleassistance inquiry 247 over a second communication module from theplurality of wireless communication modules. In certain examples, thehighest bandwidth communication module and the second communicationmodule can both comprise Long Term Evolution (LTE) communicationsmodules (e.g., from differing service providers). In variations, thehighest bandwidth communication module can comprise a LTE communicationmodule and the second communication module can comprise a differingcommunications type, such as a Wi-Fi module.

In some aspects, the communications array 255 can implement the dataprioritization scheme to prioritize transmission or streaming of theinquiry package 247 over other communications (e.g., as a highestpriority transmission). The inquiry package 247 can be transmitted orstreamed to a teleassistance system 265 over one or more networks 260.As provided herein, the teleassistance system 265 can be included as acomponent of the transport system 190 shown and described with respectto FIG. 1. Alternatively, the teleassistance system 260 can comprise oneor more separate entities, and can provide resolutions to inquirypackages 247 via dedicated computational resources, human teleassistanceoperators 266, or a combination of both. In one example, humanteleassistance operators 266 may be provided with selectable optionsgenerated programmatically by computing resources of the teleassistancesystem 265 in response to the inquiry package 247. The selectableoptions can comprise commands for the SDV 200, such as commands toignore an anomaly, wait for an anomaly to self-resolve, power cycle aparticular sensor, or maneuver around the anomaly. In variations, theselectable options can include a classification of an object, or canindicate the manner in which to treat the object, such as on a slidingscale between low risk to high risk.

According to various implementations, the SDV 200 can include atelemetry and diagnostics module 290, which can provide diagnostics data293 and telemetry data 291 to the teleassistance system 265 (e.g., aspart of the inquiry data package 247). In certain examples, thetelemetry data 291 can indicate a direction of travel and/or velocity ofthe SDV 200. Additionally, the telemetry data 291 can indicate inputsfrom the control mechanisms 295, such as braking, steering, andacceleration inputs. The diagnostic data 293 can indicate thediagnostics of certain components of the SDV 200, such as theperformance of individual sensors that comprise the sensor suite 210,engine performance, data processing performance, temperature data forthe computer stack, engine, batteries, etc., tire pressure data, and thelike.

In some aspects, the telemetry data 291 can be transmitted or streamedwith the inquiry package 247. As such, the telemetry data 291 can betreated by at least one of the data parsing engine 235, the data encoder240, or the duplication module 245 prior to transmission to theteleassistance system 265. Accordingly, the teleassistance module 225can process the telemetry data 291 by, for example, scaling thefrequency of telemetry transmissions (e.g., from 50 Hz to 5 Hz),compressing the data transmissions, or dynamically parsing the telemetrydata 291 and transmitting only data relevant to a particular anomaly.

In certain implementations, the teleassistance system 265 can make datarequests 268 to the SDV 200. In some aspects, a teleassistance operator266 can transmit a data request 268 to the SDV 200 for more information,such as telemetry data 291 or diagnostics data 293. Such data 291, 293can indicate whether a detection or object anomaly is due to a componentfault or failure. Additionally or alternatively, these data 291, 293 canprovide context to a teleassistance operator 266 in the resolution of aspecific detection inquiry 217 or object inquiry 222 included in aninquiry data package 247.

The teleassistance system 265 can generate a resolution response 269 tothe inquiry package 247, and transmit the response 269 back to the SDV200. The resolution response 269 can indicate the manner in which theSDV 200 is to respond to a given anomaly. For example, the resolutionresponse 269 can comprise a resolution plan 277 instructing the SDV 200to wait, proceed with caution, ignore the anomaly, drive to a servicestation, power cycle a component, and the like. The resolution response269 can further include information corresponding to the detectioninquiry 217 or the object inquiry 222, such as an object classification242 for an anomalous object, or fault data for a particular sensor(e.g., indicating lens debris, a misaligned sensor, etc.).

In various examples, the teleassistance module 225 can include aresolution engine 270, which can process the resolution response 269. Ifthe resolution response 269 includes an object classification 242 for ananomalous object, the resolution engine 270 can submit the objectclassification 242 to the perception interface 230, which can transmitthe classification 242 to the prediction engine 220. If the resolutionresponse includes a resolution plan 277, then the resolution engine 270can provide the plan 277 to a controller interface 275 of theteleassistance module 225, which can transmit the resolution plan 277 tothe vehicle controller 280. Thus, the vehicle controller 280 can causethe SDV 200 to proceed in accordance with the resolution plan 277,modulating inputs to the control mechanisms 295 of the SDV 200accordingly.

According to examples provided herein, the perception engine 215,prediction engine 220, and vehicle controller 280 can communicate withthe on-board teleassistance module 225 to resolve detection and objectanomalies. The teleassistance module 225 can monitor bandwidth data 257for each communications module of the communications array 255 in orderto parse, prioritize, bound, encode, compress, and/or duplicate data tobe transmitted to the teleassistance system 265. Further, in generatingan inquiry package 247, the teleassistance module 225 can encodeseparate portions of the sensor data 212 (e.g., a bounding box for avideo stream encompassing an anomalous object). The inquiry package 247can include a live sensor view of one or more sensors the SDV 200. Forexample, the live sensor view can comprise image data from a particularcamera that detects an anomalous object. Thus, the teleassistance module225 can ensure that inquiry packages 247 are reliably transmitted orstreamed to the teleassistance system 265, and can also reduce orminimize impact on bandwidth through select encoding of the sensor data212.

Self-Driving Vehicle in Operation

FIG. 3 shows an example of an autonomously controlled self-drivingvehicle utilizing sensor data to navigate an environment in accordancewith example implementations. In an example of FIG. 3, the autonomousvehicle 310 may include various sensors, such as a roof-top camera array(RTC) 322, forward-facing cameras 324 and laser rangefinders 330. Insome aspects, a data processing system 325, comprising a computer stackthat includes a combination of one or more processors, FPGAs, and/ormemory units, can be positioned in the cargo space of the vehicle 310.

According to an example, the vehicle 310 uses one or more sensor views303 (e.g., a stereoscopic or 3D image of the environment 300) to scan aroad segment on which the vehicle 310 traverses. The vehicle 310 canprocess image data or sensor data, corresponding to the sensor views 303from one or more sensors in order to detect objects that are, or maypotentially be, in the path of the vehicle 310. In an example shown, thedetected objects include a pedestrian 304 and another vehicle 327—eachof which may potentially cross into a road segment along which thevehicle 310 traverses. The vehicle 310 can use information about theroad segment and/or image data from the sensor views 303 to determinethat the road segment includes a divider 317 and an opposite lane, aswell as a sidewalk (SW) 321, and sidewalk structures such as parkingmeters (PM) 327.

The vehicle 310 may determine the location, size, and/or distance ofobjects in the environment 300 based on the sensor view 303. Forexample, the sensor views 303 may be 3D sensor images that combinesensor data from the roof-top camera array 322, front-facing cameras324, and/or laser rangefinders 330. Accordingly, the vehicle 310 mayaccurately detect the presence of objects in the environment 300,allowing the vehicle 310 to safely navigate the route while avoidingcollisions with other objects.

According to examples, the vehicle 310 may determine a probability thatone or more objects in the environment 300 will interfere or collidewith the vehicle 310 along the vehicle's current path or route. In someaspects, the vehicle 310 may selectively perform an avoidance actionbased on the probability of collision. The avoidance actions may includevelocity adjustments, lane aversion, roadway aversion (e.g., changelanes or drive further from the curb), light or horn actions, and otheractions. In some aspects, the avoidance action may run counter tocertain driving conventions and/or rules (e.g., allowing the vehicle 310to drive across center line to create space for bicyclist).

The SDV 310 can further detect certain road features that can increasethe vehicle's alertness, such as a crosswalk 315 and a traffic signal340. In the example shown in FIG. 3, the SDV 310 can identify certainfactors that can cause the vehicle 310 to enter a high alert state, suchas the pedestrian 304 being proximate to the crosswalk 315. Furthermore,the SDV 310 can identify the signal state of the traffic signal 340(e.g., green) to determine acceleration and/or braking inputs as the SDV310 approaches the intersection. At any given time, the SDV 310 candetect an anomaly—such as an indeterminate object or an issue with asensor—and query a backend teleassistance system to resolve the anomaly.

FIG. 4 shows an example self-driving vehicle executing a resolutionresponse from a backend teleassistance operator, for illustrativepurposes. In the example shown in FIG. 4, a self-driving vehicle (SDV)410 can navigate an environment and approach an intersection 407. Atruck 408 may be parked in an adjacent lane or parking space, and cancause an occlusion in the field of view 409 of one or more sensors 403of the SDV 410. For example, the occlusion may be blocking a vehicle 416that is entering the intersection 407. This occlusion can comprise adetection anomaly, and can trigger the perception and predictionresources of the SDV 410 to slow or stop the SDV 410 to ensure safety.

According to examples described herein, the SDV 410 can transmit ateleassistance inquiry indicating the occlusion caused by the parkedtruck 408 blocking its view of the intersection 407. A resolutionresponse from the teleassistance system can comprise instructions forthe SDV 410 to proceed slowly forward in a high caution state, which canenable the SDV 410 to identify the vehicle 416, and safely proceedthrough the intersection 407.

In another example of FIG. 4, the SDV 410 can detect an anomalous object432, which may cause the SDV 410 to slow or stop. As provided herein,the SDV 410 can transmit a teleassistance inquiry to the teleassistancesystem to resolve the anomalous object. In various examples, theteleassistance system can transmit a resolution response that classifiesthe anomalous object (e.g., as a low risk object), or specifies theobject (e.g., as a plastic bag). The SDV 410 may then proceed accordingto the classification of the object by ignoring the object, waiting, orproceeding with caution.

Methodology

FIGS. 5A, 5B, and 6 are flow charts describing example methods ofoperating a self-driving vehicle using teleassistance, according toexamples described herein. In the below descriptions of FIGS. 5A, 5B,and 6, reference may be made to reference characters representing likefeatures as shown and described with respect to FIGS. 1 and 2.Furthermore, the individual processes discussed with respect to FIGS.5A, 5B, and 6 may be performed by an example SDV control system 120implementing a teleassistance module 125, 225 as shown and described inFIGS. 1 and 2. Accordingly, in the below descriptions of FIGS. 5A, 5B,and 6, reference may be made interchangeably to the SDV 100, 200 ingeneral and/or the SDV control system 120 implementing a teleassistancemodule 125, 225 of FIGS. 1 and 2. Still further, it is contemplated thata particular step described either individually or as part of anadditional step can be combined with or omitted from other individuallydescribed steps of FIGS. 5A, 5B, and 6.

Referring to FIG. 5A, the control system 120 can analyze a live sensorview of the SDV 100 in order to autonomously operate the SDV 100 along agiven route 139 (500). While operating the SDV 100, the control system120 can detect an anomaly in the live sensor view (502). As describedherein, the anomaly can comprise at least one of a detection anomaly,such as an occlusion (503), or an object anomaly, such as anindeterminate object (504). Based on the detected anomaly, the controlsystem 120 can generate a teleassistance inquiry 143 for transmission toa backend teleassistance system 265 over one or more networks 260 (505).In certain implementations, the control system 120 can parse the sensordata 115 to determine a particular sensor and/or a particular portion orview field of the sensor data 115 in includes the detected anomaly(507). Additionally or alternatively, the control system 120 canprioritize the sensor data type based on the detection anomaly (509). Asan example, for object anomalies, the control system 120 can prioritizeimage data as opposed to LIDAR data or radar data for transmission tothe teleassistance system 265. As another example, for detectionanomalies (e.g., an occlusion), the control system 120 can prioritizeLIDAR data over image data.

In some implementations, the control system 120 can monitor thebandwidth data of the SDV's communication system 255 (510). Thebandwidth data can indicate the available bandwidth or transmissionperformance of each communications module (e.g., a 4G LTE modem) of theSDV 100. Based on the bandwidth data, the control system 120 can selectone or more optimal communications modules to transmit theteleassistance inquiry 143 (512), and transmit the teleassistanceinquiry 143 over the optimal communications module(s) (514).

In some aspects, the control system 120 can receive a request 268 foraddition data from the teleassistance system 265 (516). For example, therequest 268 can be for additional sensor data 115, or more detailedsensor data 115, to provide additional context to the detected anomaly(517). Additionally or alternatively, the request 268 may be fortelemetry data 291 or diagnostics data 293 for added context. Inresponse to the data request 268, the control system 120 can compile therequested data, and transmit or stream the data to the teleassistancesystem 265.

Based on the teleassistance inquiry 143, the control system 120 canreceive a resolution response 269 from the teleassistance system 265(522). The resolution response 269 can include a procedure instructionindicating the manner in which the control system 120 should proceed, orcan indicate a classified object 242 (e.g., specifying the anomalousobject as a ball, bag, cat, manhole cover, mattress, etc.) (526). Basedon the resolution response 269, the control system 120 can proceedaccordingly.

FIG. 5B is a flow chart describing a lower level method of operating anSDV 100, 200 using a teleassistance module 125, 225, as provided herein.Referring to FIG. 5B, the control system 120 can analyze the live sensorview to autonomously operate the SDV 100 along a given route 139 (530).While operating the SDV 100, the control system 120 can transmit andreceive data from a backend transport system 190 that manages routingfor a fleet of SDVs operating throughout the given region (532). Thesedata transmissions can include periodic location and/or data pings 121to the transport system 190 (534), and can further includetransportation instructions and/or route updates (536).

In various examples, the control system 120 can identify a detectionanomaly in the live sensor view (540). The detection anomaly canindicate an underlying issue in the sensor data, such as debris on alens, a misalignment, an occlusion in the field of view of one or moresensors, and the like. In treating the detection anomaly, the controlsystem 120 can determine whether a safety threshold or detection hazardthreshold is met (542). For example, the control system 120 candetermine a value for the sensor data quality to determine whether thesensor data is usable, or can determine that a collision probability isincreased above a certain tolerance range due to the detection anomaly.

If the safety threshold is still met (543), then the control system 120can proceed with caution to resolve the detection anomaly and continuealong the given route 139 (546). However, if the safety threshold is notmet (544), then the control system 120 can generate a teleassistanceinquiry 143 to resolve the detection anomaly (548). In generating theteleassistance inquiry 143, the control system 120 can prioritize datafrom certain sensors, such as sensors having a relevant field of viewthat covers the detection anomaly, or a sensor experiencing detectionissues (550). In variations, the control system 120 can have a defaultoption for detection anomalies, in which the control system 120automatically prioritizes LIDAR data (e.g., to resolve an occlusion)(552).

According to some examples, the control system 120 can identify anobject anomaly in the live sensor view—or an indeterminate object thatthe perception/prediction engine 140 is unable to classify (554). Intreating object anomalies, the control system 120 can determine whethera certainty threshold is met for the object (556). For example, theperception/prediction engine 140 can identify an object in the sensordata 115 that is in the path of the SDV 100 and attempt to identify orclassify the object. In this example, if a 95% threshold is not met forthe object in the path of the SDV 100, then the control system 120 cangenerate a teleassistance inquiry 143 for the indeterminate object.

Accordingly, if the certainty threshold is met (557), then the controlsystem 120 can proceed with caution along the route 139 having resolvedthe object anomaly. However, if the certainty threshold is not met(558), then the control system 120 can generate a teleassistance inquiry143 for the indeterminate object (560). In generating the teleassistanceinquiry 143, the control system 120 can also prioritize data fromindividual sensors, such as an individual camera having theindeterminate object in its field of view (562). Additionally oralternative, the control system 120 can, by default, prioritize cameradata or video streams over other data types for object anomalies (564).

Whether the detected or identified anomaly comprises a detection anomalyor an object anomaly, the control system 120 can also prioritize theteleassistance inquiry 143 for transmission over the SDV'scommunications system 255 (566). In some aspects, the control system 120can do so by determining bandwidth information for each communicationmodule of the communication system 255 (568), and selecting one or moreoptimal communications modules for transmission (570). Additionally oralternatively, the control system 120 can determine whether an availablebandwidth threshold is met for the selected communications module(s) orthe communications system 255 in general. If so (573), then the controlsystem 120 can transmit or stream the teleassistance inquiry 143 to theteleassistance system 265 over the selection communications module(s)(578). However, if not (574), then the control system 120 can duplicatethe teleassistance inquiry 143 or a portion thereof (e.g., a portion ofthe sensor data) for transmission over one or more separatecommunications module (576), and transmit the teleassistance inquiry 143and duplicated data over the communications modules (578).

According to examples, the control system 120 may then receive aresolution response 269 from the teleassistance system 265 (580). Asdescribed, the resolution response 269 can include procedureinstructions indicating the manner in which the SDV 100 is to proceed toresolve a detection anomaly (582), or an object classification 242specifying or classifying an indeterminate object (584). Based on theresolution response 269, the control system 120 can proceed along theroute or an alternative route (586).

FIG. 6 is a flow chart describing an example method of resolving adetected anomaly by a self-driving vehicle, in accordance with examplesdescribed herein. One or more processes discussed with respect to FIG. 6may be performed in conjunction with, or as additional processes of thesteps described in connection with the flow charts of FIGS. 5A and 5B.Referring to FIG. 6, the control system 120 of the SDV 100 can identifyan object anomaly in the live sensor view of the SDV 100 (600). In someexamples, the control system 120 can select one or more cameras having afield of view that includes the object anomaly, and bound the objectanomaly in the image data of the selected one or more cameras (605). Asused herein, the image data can comprise live video data from the one ormore cameras, and the control system 120 can bound the object anomalydynamically as the SDV 100 moves. Furthermore, the object anomaly cancomprise an indeterminate object in which the perception and/orprediction resources of the control system 120 are unable to classify orspecify with certainty (e.g., above a certainty threshold).

By bounding the object anomaly, the control system 120 can then encodethe image data for transmission over one or more communications modulesof the SDV 100 (610). In various implementations, the control system 120can encode the bounded area (e.g., a bounding box) including the objectanomaly at a first bitrate (612), and encode the remaining image data ata second bitrate (614), for example, in order to decrease or minimizeimpact on bandwidth usage. The control system 120 may then transmit orstream the image data to a remote teleassistance system 265 over one ormore networks 260 (615).

In some aspects, the control system 120 can receive a data request 268for additional data, such as additional image data, radar data, or LIDARdata (620). In response to the data request, 268, the control system 120can compile the requested data and transmit or stream the data to theteleassistance system 265 over the one or more networks 260 (625).According to examples provided herein, the control system 120 canreceive data specifying or classifying the object anomaly from theteleassistance system 265 (630). Based on the determined object, thecontrol system 120 can cause the SDV 100 to proceed along the currentroute 139 or an alternate route accordingly (635). For example, thecontrol system 120 can ignore the object if the object comprises a lowrisk object, such as a plastic bag. As another example, the controlsystem 120 can wait or carefully maneuver around the object if theobject could cause damage to the SDV 100 or if the object comprises ahigh risk object, such as an animal or a baby stroller.

Hardware Diagrams

FIG. 7 is a block diagram illustrating a computer system upon whichexample SDV processing systems described herein may be implemented. Thecomputer system 700 can be implemented using a number of processingresources 710, which can comprise processors 711, field programmablegate arrays (FPGAs) 713. In some aspects, any number of processors 711and/or FPGAs 713 of the computer system 700 can be utilized ascomponents of a neural network array 712 implementing a machine learningmodel and utilizing road network maps stored in memory 761 of thecomputer system 700. In the context of FIGS. 1 and 2, various aspectsand components of the control system 120, 205, route planning engine160, 285, and perception and prediction engines 140, 215, 220, can beimplemented using one or more components of the computer system 700shown in FIG. 7.

According to some examples, the computer system 700 may be implementedwithin an autonomous vehicle or self-driving vehicle (SDV) with softwareand hardware resources such as described with examples of FIGS. 1 and 2.In an example shown, the computer system 700 can be distributedspatially into various regions of the SDV, with various aspectsintegrated with other components of the SDV itself. For example, theprocessing resources 710 and/or memory resources 760 can be provided ina cargo space of the SDV. The various processing resources 710 of thecomputer system 700 can also execute control instructions 762 usingmicroprocessors 711, FPGAs 713, a neural network array 712, or anycombination of the same.

In an example of FIG. 7, the computer system 700 can include acommunication interface 750 that can enable communications over anetwork 780. In one implementation, the communication interface 750 canalso provide a data bus or other local links to electro-mechanicalinterfaces of the vehicle, such as wireless or wired links to and fromcontrol mechanisms 720 (e.g., via a control interface 722), sensorsystems 730, and can further provide a network link to a backendtransport management system (implemented on one or more datacenters)over one or more networks 780. For example, the processing resources 710can receive a destination 782 over the one or more networks 780, or viaa local user interface of the SDV.

The memory resources 760 can include, for example, main memory 761, aread-only memory (ROM) 767, storage device, and cache resources. Themain memory 761 of memory resources 760 can include random access memory(RAM) 768 or other dynamic storage device, for storing information andinstructions which are executable by the processing resources 710 of thecomputer system 700. The processing resources 710 can executeinstructions for processing information stored with the main memory 761of the memory resources 760. The main memory 761 can also storetemporary variables or other intermediate information which can be usedduring execution of instructions by the processing resources 710. Thememory resources 760 can also include ROM 767 or other static storagedevice for storing static information and instructions for theprocessing resources 710. The memory resources 760 can also includeother forms of memory devices and components, such as a magnetic disk oroptical disk, for purpose of storing information and instructions foruse by the processing resources 710. The computer system 700 can furtherbe implemented using any combination of volatile and/or non-volatilememory, such as flash memory, PROM, EPROM, EEPROM (e.g., storingfirmware 769), DRAM, cache resources, hard disk drives, and/or solidstate drives.

The memory 761 may also store localization maps 764 in which theprocessing resources 710—executing the control instructions762—continuously compare to sensor data 732 from the various sensorsystems 730 of the SDV. Execution of the control instructions 762 cancause the processing resources 710 to generate control commands 715 inorder to autonomously operate the SDV's acceleration 722, braking 724,steering 726, and signaling systems 728 (collectively, the controlmechanisms 720). Thus, in executing the control instructions 762, theprocessing resources 710 can receive sensor data 732 from the sensorsystems 730, dynamically compare the sensor data 732 to a currentlocalization map 764, and generate control commands 715 for operativecontrol over the acceleration, steering, and braking of the SDV. Theprocessing resources 710 may then transmit the control commands 715 toone or more control interfaces 722 of the control mechanisms 720 toautonomously operate the SDV through road traffic on roads and highways,as described throughout the present disclosure.

The memory 761 may also store teleassistance instructions 766 that theprocessing resources 710 can execute to identify detection or objectanomalies, and transmit teleassistance inquiries 719 to a backendtransport system or teleassistance system over the network 780, andreceive a resolution response 784 in return. Execution of theinstructions 762, 764, 766 can cause the processing resources 710 toprocess the resolution response 784 accordingly to resolve the detectedanomaly. Thereafter, the processing resources 710 can generate controlcommands 715 to cause the control mechanisms 720 to autonomously operatethe SDV along the current route or an alternate route accordingly.

FIG. 8 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented. A computer system 800 canbe implemented on, for example, a server or combination of servers. Forexample, the computer system 800 may be implemented as part of a networkservice for providing transportation services. In the context of FIGS. 1and 2, the transport system 190 or teleassistance system 265 may beimplemented using a computer system 800 such as described by FIG. 8.

In one implementation, the computer system 800 includes processingresources 810, a main memory 820, a read-only memory (ROM) 830, astorage device 840, and a communication interface 850. The computersystem 800 includes at least one processor 810 for processinginformation stored in the main memory 820, such as provided by a randomaccess memory (RAM) or other dynamic storage device, for storinginformation and instructions which are executable by the processor 810.The main memory 820 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by the processor 810. The computer system 800 may also includethe ROM 830 or other static storage device for storing staticinformation and instructions for the processor 810. A storage device840, such as a magnetic disk or optical disk, is provided for storinginformation and instructions.

The communication interface 850 enables the computer system 800 tocommunicate over one or more networks 880 (e.g., cellular network)through use of the network link (wireless or wired). Using the networklink, the computer system 800 can communicate with one or more computingdevices, one or more servers, and/or one or more self-driving vehicles.In accordance with examples, the computer system 800 can receive pick-uprequests 882 from mobile computing devices of individual users. Theexecutable instructions stored in the memory 830 can include selectioninstructions 822, which the processor 810 executes to select an optimaldriver or SDV to service the pick-up request 882. In doing so, thecomputer system can receive vehicle locations of drivers and SDVsoperating throughout the given region, and the processor can execute theselection instructions 822 to select an optimal driver or SDV from a setof available vehicles, and transmit a transport invitation 852 to enablethe driver to accept or decline the ride service offer, or to instructthe matched SDV to rendezvous with the requesting user.

The executable instructions stored in the memory 820 can also includeteleassistance instructions 824, which enables the computer system 800to receive teleassistance inquiries 884 from SDVs operating throughoutthe given region. In some aspects, execution of the teleassistanceinstructions 824 can cause the computer system 800 to automaticallygenerate a resolution response 856. In addition or as a variation, thecomputer system 800 can transmit the teleassistance inquiries 884 overone or more teleassistance interfaces 833 to human teleassistanceoperators 835, which can cause the resolution responses 856 to begenerated and then transmitted back to the SDVs in order to resolvedetected anomalies.

By way of example, the instructions and data stored in the memory 820can be executed by the processor 810 to implement an example transportsystem 190 or teleassistance system 265 of FIGS. 1 and 2. In performingthe operations, the processor 810 can receive pick-up requests 882 andsubmit transport invitations 852 to facilitate the servicing of therequests 882. Furthermore, the processor 810 can receive teleassistanceinquiries 884 from SDVs operating throughout the given region, andprovide resolution responses 856 accordingly.

The processor 810 is configured with software and/or other logic toperform one or more processes, steps and other functions described withimplementations, such as described with respect to FIGS. 1-6, andelsewhere in the present application. Examples described herein arerelated to the use of the computer system 800 for implementing thetechniques described herein. According to one example, those techniquesare performed by the computer system 800 in response to the processor810 executing one or more sequences of one or more instructionscontained in the main memory 820. Such instructions may be read into themain memory 820 from another machine-readable medium, such as thestorage device 840. Execution of the sequences of instructions containedin the main memory 820 causes the processor 810 to perform the processsteps described herein. In alternative implementations, hard-wiredcircuitry may be used in place of or in combination with softwareinstructions to implement examples described herein. Thus, the examplesdescribed are not limited to any specific combination of hardwarecircuitry and software.

It is contemplated for examples described herein to extend to individualelements and concepts described herein, independently of other concepts,ideas or systems, as well as for examples to include combinations ofelements recited anywhere in this application. Although examples aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the concepts are not limited to thoseprecise examples. As such, many modifications and variations will beapparent to practitioners skilled in this art. Accordingly, it isintended that the scope of the concepts be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anexample can be combined with other individually described features, orparts of other examples, even if the other features and examples make nomention of the particular feature. Thus, the absence of describingcombinations should not preclude claiming rights to such combinations.

What is claimed is:
 1. A system comprising: a network communicationinterface; one or more processors; and a memory storing instructionsthat, when executed by the one or more processors, cause the system to:analyze a live sensor view generated by a set of sensors of aself-driving vehicle (SDV) in accordance with a safety threshold to (i)detect objects of interest along a route, and (ii) classify eachrespective object of the detected objects of interest; for a particularobject of the detected objects of interest, determine that the safetythreshold is not met; in response to determining that the safetythreshold is not met for the particular object, transmit, using thenetwork communication interface, a first teleassistance inquiry to abackend computing system, the first teleassistance inquiry comprisingLIDAR data generated by a LIDAR device of the SDV, wherein the set ofsensors of the SDV includes sensors of the LIDAR device; determine thata classification of one or more of the detected objects of interest doesnot satisfy a certainty threshold; in response to determining that theclassification of the one or more of the detected objects of interestdoes not satisfy the certainty threshold, transmit, using the networkcommunication interface, a second teleassistance inquiry to the backendcomputing system, the second teleassistance inquiry comprising imagedata generated by one or more cameras of the SDV, the image dataincluding the one or more of the detected objects of interest, whereinthe set of sensors of the SDV includes sensors of the one or morecameras, and wherein the second teleassistance inquiry is different fromthe first teleassistance inquiry; for each of the first and the secondteleassistance inquiries, receive a resolution response from the backendcomputing system; and provide a control system of the SDV with datacorresponding to the resolution response to enable the SDV to proceed inaccordance with the resolution response.
 2. The system of claim 1,wherein the safety threshold not being met corresponds to an obstructionin the live sensor view.
 3. The system of claim 1, wherein the executedinstructions cause the system to transmit LIDAR data and not image datawhen the safety threshold is not met.
 4. The system of claim 1, whereinthe executed instructions cause the system to transmit image data andnot LIDAR data when the certainty threshold is not met.
 5. The system ofclaim 1, wherein the executed instructions further cause the system to:monitor bandwidth of each of a plurality of wireless communicationmodules of the network communication interface to determine a highestbandwidth communication module; wherein the executed instructions causethe system to transmit each of the first and the second teleassistanceinquiries to the backend computing system using the highest bandwidthcommunication module.
 6. The system of claim 5, wherein the executedinstructions further cause the system to: duplicate transmission of eachof the first and the second teleassistance inquiries by transmitting thefirst and the second teleassistance inquiries over a secondcommunication module from the plurality of wireless communicationmodules.
 7. The system of claim 6, wherein the highest bandwidthcommunication module and the second communication module both compriseLong Term Evolution (LTE) communications modules.
 8. The system of claim6, wherein the highest bandwidth communication module comprises a LTEcommunication module and the second communication module comprises aWi-Fi module.
 9. The system of claim 1, wherein the first and the secondteleassistance inquiries include telemetry data for the SDV.
 10. Thesystem of claim 1, wherein the first and the second teleassistanceinquiries include localization information for the SDV.
 11. The systemof claim 1, wherein the first and the second teleassistance inquiriesinclude diagnostics data for the SDV.
 12. A non-transitory computerreadable medium storing instructions that, when executed by one or moreprocessors, cause the one or more processors to: analyze a live sensorview generated by a set of sensors of a self-driving vehicle (SDV) inaccordance with a safety threshold to (i) detect objects of interestalong a route, and (ii) classify each respective object of the detectedobjects of interest; for a particular object of the detected objects ofinterest, determine that the safety threshold is not met; in response todetermining that the safety threshold is not met for the particularobject, transmit, using a network communication interface, a firstteleassistance inquiry to a backend computing system, the firstteleassistance inquiry comprising LIDAR data generated by a LIDAR deviceof the SDV, wherein the set of sensors of the SDV includes sensors ofthe LIDAR device; determine that a classification of one or more of thedetected objects of interest does not satisfy a certainty threshold; inresponse to determining that the classification of the one or more ofthe detected objects of interest does not satisfy the certaintythreshold, transmit, using the network communication interface, a secondteleassistance inquiry to the backend computing system, the secondteleassistance inquiry comprising image data generated by one or morecameras of the SDV, the image data including the one or more detectedobjects of interest, wherein the set of sensors of the SDV includessensors of the one or more cameras, and wherein the secondteleassistance inquiry is different from the first teleassistanceinquiry; for each of the first and the second teleassistance inquiries,receive a resolution response from the backend computing system; andprovide a control system of the SDV with data corresponding to theresolution response to enable the SDV to proceed in accordance with theresolution response.
 13. The non-transitory computer readable medium ofclaim 12, wherein the safety threshold not being met corresponds to anobstruction in the live sensor view.
 14. The non-transitory computerreadable medium of claim 12, wherein the executed instructions cause thesystem to transmit LIDAR data and not image data when the safetythreshold is not met.
 15. The non-transitory computer readable medium ofclaim 12, wherein the executed instructions cause the system to transmitimage data and not LIDAR data when the certainty threshold is not met.16. The non-transitory computer readable medium of claim 12, wherein theexecuted instructions further cause the system to monitor bandwidth ofeach of a plurality of wireless communication modules of the networkcommunication interface to determine a highest bandwidth communicationmodule; wherein the executed instructions cause the system to transmiteach of the first and the second teleassistance inquiries to the backendcomputing system using the highest bandwidth communication module. 17.The non-transitory computer readable medium of claim 16, wherein theexecuted instructions further cause the system to: duplicatetransmission of each of the first and the second teleassistanceinquiries by transmitting the first and the second teleassistanceinquiries over a second communication module from the plurality ofwireless communication modules.
 18. The non-transitory computer readablemedium of claim 17, wherein the highest bandwidth communication moduleand the second communication module both comprise Long Term Evolution(LTE) communications modules.
 19. A computer-implemented methodperformed by one or more processors and comprising: analyzing a livesensor view generated by a set of sensors of a self-driving vehicle(SDV) in accordance with a safety threshold to (i) detect objects ofinterest along a route, and (ii) classify each respective object of thedetected objects of interest; for a particular object of the detectedobjects of interest, determining that the safety threshold is not met;in response to determining that the safety threshold is not met for theparticular object, transmitting, using a network communicationinterface, a first teleassistance inquiry to a backend computing system,the first teleassistance inquiry comprising LIDAR data generated by aLIDAR device of the SDV, wherein the set of sensors of the SDV includessensors of the LIDAR device; determining that a classification of one ormore of the detected objects of interest does not satisfy a certaintythreshold; in response to determining that the classification of the oneor more of the detected objects of interest does not satisfy thecertainty threshold, transmitting, using the network communicationinterface, a second teleassistance inquiry to the backend computingsystem, the second teleassistance inquiry comprising image datagenerated by one or more cameras of the SDV, the image data includingthe one or more detected objects of interest, wherein the set of sensorsof the SDV includes sensors of the one or more cameras, and wherein thesecond teleassistance inquiry is different from the first teleassistanceinquiry; for each of the first and the second teleassistance inquiries,receiving a resolution response from the backend computing system; andproviding a control system of the SDV with data corresponding to theresolution response to enable the SDV to proceed in accordance with theresolution response.